Image processor and monitoring system

ABSTRACT

The image processor of the present invention generates a more natural synthesized image from camera images taken with a plurality of cameras capturing the surroundings of a vehicle. A parameter storage section stores a plurality of image synthesis parameter groups representing the correspondence between the camera images and the synthesized image and having different spatial or temporal resolution relations. A parameter selection section selects the image synthesis parameter group according to the output of a vehicle motion detection section for detecting the motion of the vehicle such as the driving speed and direction. An image synthesis section generates the synthesized image from the camera images according to the selected image synthesis parameter group.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to an image processing techniquefor generating a synthesized image from images taken with a plurality ofcameras. In particular, the present invention relates to a techniquethat can be effectively applied to monitoring systems used as an assistin ensuring safety driving of a vehicle and the like.

[0002] In recent years, with widespread use of car-mounted displays andcost reduction in video equipment such as cameras, devices formonitoring the surroundings of a vehicle with cameras to assist safetydriving have been commercialized and are now becoming popular.

[0003] As an example of such conventional vehicle surroundingsmonitoring devices, the following construction is known. That is, acamera is placed at a position on a vehicle where an image of a blindarea for the driver, such as the rear of the vehicle, can be taken withthe camera, and the image taken (camera image) is shown on a display formonitoring.

[0004] For example, Japanese Laid-Open Patent Publication No. 58-110334discloses the following construction. A plurality of cameras are placedat different positions on a vehicle. A plurality of camera images takenwith the cameras are modified and synthesized, to generate a synthesizedimage as is viewed from the above of the vehicle with the vehicle in thecenter and display the synthesized image on an in-car display. Usingthis vehicle surroundings monitoring device, the driver sitting on thedriver's seat can monitor on the display a blind area of which directview is not possible. This contributes to avoiding an accident thatwould otherwise have occurred, and also facilitating the driving.

[0005] [Problems to be Solved]

[0006] However, the inventors of the present invention have found fromexperiments and examinations that the conventional construction has thefollowing problems.

[0007]FIGS. 22A to 22C illustrate examples of image synthesis performedwhen a vehicle is moving. In the case of using cameras for interlacedscanning, an input image as shown in FIG. 22A is obtained when thevehicle is moving. That is, every other horizontal line is displacedresulting in comb-shaped appearance. This input image is notparticularly strange as long as it is displayed as it is on a displayfor interlaced scanning. However, when this input image is modified forsynthesis, the comb-shaped displacement is also modified during theimage modification as shown in FIG. 22B, and this causes a lag betweenthe image-taking timing and the displaying timing. The resultant imagemakes the observer feel strange. That is, so-called interlace noiseappears on the synthesized image more significantly.

[0008] The unnaturalness as described above does not appear when aone-field synthesized image is generated from one-field input images.However, the resolution of an input image is low in this per-fieldsynthesis, compared with per-frame synthesis. Therefore, when an imageof two fields is observed as a frame image as shown in FIG. 22C,comb-shaped displacement occurs due to an error in quantization ofcoordinate values even when the image is still. This deteriorates theimage quality, especially in the area of a synthesized image where aninput image has been enlarged, when the vehicle is standing still.

[0009] That is, as the first problem, when a one-frame synthesized imageis generated from one-frame (two-field) input images, the synthesizedimage is unnatural when the vehicle is moving and when a moving image istaken. When a one-field synthesized image is generated from one-fieldinput images, the resolution of the synthesized image is low.

[0010] As the second problem, in modification of a camera image, theinput image may be contracted depending on the composition of asynthesized image. As a result, aliasing distortion may partially begenerated in the synthesized image.

[0011] As the third problem, the junctures of input images on asynthesized image may be unnatural. For example, because white balancingand the like are adjusted separately for the respective cameras, thebrightness and the tint may fail to match at the junctures as shown inFIG. 23A. Also, in some image synthesis methods, the junctures of imagesmay be displaced due to calculation error, erroneous detection of theposition of an object, and the like, as shown in FIG. 23B.

SUMMARY OF THE INVENTION

[0012] An object of the present invention is providing an imageprocessor for generating a synthesized image from a plurality of cameraimages, which can generate a more natural synthesized image comparedwith the conventional processors.

[0013] Specifically, the image processor of the present invention, whichreceives camera images taken with a plurality of cameras capturing thesurroundings of a vehicle and generates a synthesized image from thecamera images, includes a parameter generation section constructed to beable to generate a plurality of image synthesis parameter groups eachrepresenting the correspondence between the camera images and thesynthesized image and having different spatial or temporal resolutionrelations, wherein the synthesized image is generated from the cameraimages according to the image synthesis parameter group generated by theparameter generation section, and the parameter generation sectionswitches the image synthesis parameter group to be generated accordingto an output of a vehicle motion detection section for detecting themotion of the vehicle.

[0014] According to the invention described above, it is possible toswitch the image synthesis parameter group used for generation of asynthesized image according to the motion of the vehicle. This makes itpossible to reduce the unnaturalness of the synthesized image observedwhen the vehicle is moving and when an image of a moving object istaken, and also prevent the reduction in resolution observed when thevehicle is standing still.

[0015] In the image processor of the present invention, the parametergeneration section preferably includes: a parameter storage section forstoring the plurality of image synthesis parameter groups; and aparameter selection section for selecting at least one among theplurality of image synthesis parameter groups stored in the parameterstorage section according to the output of the vehicle motion detectionsection.

[0016] In the image processor of the present invention, preferably, thecamera images are interlaced images, and the plurality of imagesynthesis parameter groups include at least a frame-base image synthesisparameter group and a field-base image synthesis parameter group.Preferably, the parameter generation section generates the field-baseimage synthesis parameter group when the motion of the vehicle detectedby the vehicle motion detection section is relatively fast, andgenerates the frame-base image synthesis parameter group when the motionof the vehicle is relatively slow.

[0017] In the image processor of the present invention, the vehiclemotion detection section preferably detects the motion of the vehiclefrom the camera images.

[0018] In the image processor of the present invention, preferably, theplurality of cameras are constructed to be able to switch a capturepattern according to an input switch signal, and the parametergeneration section sends the switch signal to the cameras, together withgenerating the image synthesis parameter group, according to the outputof the vehicle motion detection section, to switch the capture patternsof the cameras. Preferably, the parameter generation section selects theimage synthesis parameter group and switches the capture patterns of thecameras according to an output of a vehicle status detection section fordetecting brightness of the surroundings of the vehicle, in addition tothe output of the vehicle motion detection section.

[0019] In the image processor of the present invention, preferably, theparameter generation section generates the image synthesis parametergroup according to an output of a vehicle status detection section fordetecting the vehicle status such as an operation by a driver of thevehicle or whether or not an obstacle exists in the surroundings of thevehicle, in addition to the output of the vehicle motion detectionsection.

[0020] The monitoring system of the present invention includes the imageprocessor described above as an image processing section.

[0021] Alternatively, the image processor of the present invention,which receives camera images taken with a plurality of cameras capturingthe surroundings of a vehicle and generates a synthesized image from thecamera images, includes: a parameter storage section for storing aplurality of sets of an image synthesis parameter group representing thecorrespondence between the camera images and the synthesized image and afilter parameter group corresponding to the image synthesis parametergroup; a parameter selection section for selecting at least one amongthe plurality of sets of the image synthesis parameter group and thefilter parameter group stored by the parameter storage section accordingto an output of a vehicle motion detection section for detecting themotion of the vehicle and an output of a vehicle status detectionsection for detecting the status of the vehicle such as an operation bya driver of the vehicle or whether or not an obstacle exists in thesurroundings of the vehicle; and a filtering section for performingfrequency band limitation filtering for the camera images according tothe filter parameter group of the set selected by the parameterselection section, wherein the synthesized image is generated from thecamera images filtered by the filtering section according to the imagesynthesis parameter group of the set selected by the parameter selectionsection.

[0022] According to the invention described above, the filter parametergroup is selected according to the motion and status of the vehicle, andthe camera images are subjected to frequency band limitation filteringaccording to the selected filter parameter group. This effectivelysuppresses the aliasing distortion in the synthesized image.

[0023] In the image processor of the present invention described above,the filter parameter group preferably includes filtering setting datafor each pixel position of the camera image.

[0024] The monitoring system of the present invention includes the imageprocessor described above as an image processing section.

[0025] Alternatively, the image processor of the present invention,which receives camera images taken with a plurality of cameras capturingthe surroundings of a vehicle and generates a synthesized image from thecamera images, includes: a brightness correction parameter calculationsection for calculating brightness correction parameters for correctingthe brightness and tint of the camera images; and a brightnesscorrection section for correcting the brightness and tint of the cameraimages using the brightness correction parameters calculated by thebrightness correction parameter calculation section, wherein thesynthesized image is generated from the plurality of camera imagessubjected to brightness correction by the brightness correction sectionaccording to an image synthesis parameter group representing thecorrespondence between the camera images and the synthesized image, theimage synthesis parameter group includes overlap area data for anoverlap area on the synthesized image in which coverages of theplurality of cameras overlap, the overlap area data indicatingcoordinates of pixels in camera images corresponding to the overlaparea, and the brightness correction parameter calculation sectionreceives the overlap area data and calculates the brightness correctionparameters using brightness and tint data for the pixels in the cameraimages corresponding to the overlap area indicated by the overlap areadata.

[0026] According to the invention described above, brightness correctionparameters are calculated using information on the brightness and tintof the camera images corresponding to the overlap area on thesynthesized image, and brightness correction is performed using thebrightness correction parameters. This reduces the unnaturalness of thejuncture on the synthesized image.

[0027] In the image processor of the present invention described above,preferably, the brightness correction parameter calculation sectionperforms statistical processing on the brightness in the overlap areafor the camera images corresponding to the overlap area, and calculatesthe brightness correction parameters based on the processing results.

[0028] In the image processor of the present invention described above,when a plurality of overlap areas exist, the brightness correctionparameter calculation section preferably sets priorities to the overlapareas to be considered during the calculation of the brightnesscorrection parameters according to an output of a vehicle motiondetection section for detecting the motion of the vehicle.

[0029] In the image processor of the present invention described above,the brightness correction section is preferably incorporated in thecameras.

[0030] The monitoring system of the present invention includes the imageprocessor described above as an image processing section.

[0031] Alternatively, the image processor of the present inventionreceives camera images taken with a plurality of cameras capturing thesurroundings of a vehicle and generates a synthesized image from thecamera images, wherein, in an overlap area in which coverages of aplurality of cameras overlap on the synthesized image, a camera imageused for generation of the synthesized image is selected among cameraimages from the plurality of cameras according to an output of a vehiclemotion detection section for detecting the motion of the vehicle or anoutput of a vehicle status detection section for detecting the status ofthe vehicle such as an operation by a driver of the vehicle or whetheror not an obstacle exists in the surroundings of the vehicle.

[0032] Alternatively, the image processor of the present inventionreceives camera images taken with a plurality of cameras capturing thesurroundings of a vehicle and generates a synthesized image from thecamera images, wherein, in an overlap area in which overages of aplurality of cameras overlap on the synthesized image, weights to cameraimages from the plurality of cameras are set according to an output of avehicle motion detection section for detecting the motion of the vehicleor an output of a vehicle status detection section for detecting thestatus of the vehicle such as an operation by a driver of the vehicle orwhether or not an obstacle exists in the surroundings of the vehicle.

[0033] In the image processor of the present invention described above,the processor includes: an image synthesis parameter group associatingpixels in the synthesized image with pixels in the camera images; and atable representing the correspondence between a weight reference numberand a combination of sets of weighting information, wherein a portion ofthe image synthesis parameter group corresponding to the overlap areaholds any of the weight reference numbers shown in the table.

[0034] The monitoring system of the present invention includes the imageprocessor described above as an image processing section.

[0035] Alternatively, the image processor of the present inventionreceives camera images taken with a plurality of cameras capturing thesurroundings of a vehicle and generating a synthesized image from thecamera images, wherein the image processor includes an image synthesisparameter group associating pixels in the synthesized image with pixelsin the camera images, and in an overlap area in which coverages of theplurality of cameras overlap on the synthesized image, the imagesynthesis parameter group includes weights set to camera images from theplurality of cameras represented by a dithering method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0036]FIG. 1 is a structural view of a vehicle surroundings monitoringsystem of EMBODIMENT 1 of the present invention.

[0037]FIGS. 2A to 2C are views for description of image synthesis usingan image synthesis parameter group.

[0038]FIG. 3 shows an example of selection of image synthesis parametergroups in EMBODIMENT 1 of the present invention.

[0039]FIG. 4 shows an example of selection of image synthesis parametergroups and capture patterns in EMBODIMENT 1 of the present invention.

[0040]FIG. 5 shows an example of selection of image synthesis parametergroups and capture patterns in the case of using an output of abrightness detection means in EMBODIMENT 1 of the present invention.

[0041]FIG. 6 shows an example of selection of image synthesis parametergroups in the case of using outputs of a switch detection means and anobstacle detection means in EMBODIMENT 1 of the present invention.

[0042]FIGS. 7A and 7B show exemplary compositions of a synthesizedimage.

[0043]FIG. 8 is a structural view of a vehicle surroundings monitoringsystem of EMBODIMENT 2 of the present invention.

[0044]FIG. 9 is a view showing a filter parameter group.

[0045]FIG. 10 shows an example of selection of image synthesis parametergroups and filter parameter groups in EMBODIMENT 2 of the presentinvention.

[0046]FIGS. 11A and 11B are views showing filter parameter groups.

[0047]FIG. 12 is a structural view of a vehicle surroundings monitoringsystem of EMBODIMENT 3 of the present invention.

[0048]FIGS. 13A to 13D are views showing an image synthesis parametergroup for generating a synthesized image from a plurality of cameraimages.

[0049]FIGS. 14A and 14B are views showing a synthesized image includingan overlap area and an image synthesis parameter group for generatingthe synthesized image.

[0050]FIG. 15 is a view showing the positions at which cameras andobstacle sensors are placed in EMBODIMENT 4 of the present invention.

[0051]FIGS. 16A to 16C are views showing a synthesized image and imagesynthesis parameter groups for generating the synthesized image.

[0052]FIGS. 17A to 17C are views showing examples of selection of animage synthesis parameter group in EMBODIMENT 4 of the presentinvention.

[0053]FIG. 18 is a view showing another image synthesis parameter groupin EMBODIMENT 4 of the present invention.

[0054]FIG. 19 is a view showing yet another exemplary image synthesisparameter group in EMBODIMENT 4 of the present invention.

[0055]FIGS. 20A and 20B are views showing other image synthesisparameter groups in EMBODIMENT 4 of the present invention.

[0056]FIG. 21 is a view showing another construction for implementingthe present invention.

[0057]FIGS. 22A to 22C are views for description of conventionalproblems.

[0058]FIGS. 23A and 23B are views for description of conventionalproblems.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0059] Hereinafter, preferred embodiments of the present invention willbe described with reference to the accompanying drawings.

EMBODIMENT 1

[0060]FIG. 1 illustrates a construction of a vehicle surroundingsmonitoring system of EMBODIMENT 1 of the present invention. Referring toFIG. 1, cameras 101 take images, A/D converters 102 digitalize analogimage signals, and frame memories 103 temporarily hold the digitalizedinput images. Each of the frame memories 103 have a so-calleddouble-buffer structure, which temporarily holds immediately precedingone-frame image data while continuing to store the images output fromthe camera 1. In other words, the frame memories 103 are constructed topermit read of arbitrary pixel data in the immediately precedingone-frame image in response to a read request from an image synthesissection 104.

[0061] A vehicle motion detection section 107 detects motion of avehicle. A parameter storage section 109 stores in advance a pluralityof image synthesis parameter groups representing the correspondencebetween camera images and a synthesized image. A parameter selectionsection 108 selects one of the plurality of image synthesis parametergroups stored in the parameter storage section 109 according to themotion of the vehicle detected by the vehicle motion detection section107. The image synthesis section 104: sequentially reads images from theframe memories 103 according to the image synthesis parameter groupselected by the parameter selection section 108 to combine the imagesand output a synthesized image. A D/A converter 105 converts thesynthesized image to analog signals, and a display 106 displays theanalog-converted synthesized image.

[0062] The A/D converters 102, the frame memories 103, the imagesynthesis section 104, the D/A converter 105, the parameter selectionsection 108, and the parameter storage section 109 constitute an imageprocessor or an image processing section. The parameter selectionsection 108 and the parameter storage section 109 constitute a parametergeneration section.

[0063] In this embodiment, assume that images handled by the cameras 101and the display 106 are interlaced scanning images. Also assume that thevehicle motion detection section 107 detects the rotational speed of thewheels of the vehicle, the driving direction, and the like as the motionof the vehicle, from signals sent from sensors placed on an axle of awheel, the shift lever, and the like.

[0064] The system also includes a vehicle status detection section 200,which will be described later together with the usage thereof.

[0065] Hereinafter, the operation of the vehicle surroundings monitoringsystem with the above construction will be described.

[0066]FIGS. 2A to 2C illustrate how to combine images using an imagesynthesis parameter group. FIG. 2A illustrates an example of a cameraimage that is a one-frame image from a camera 1 placed as shown in FIG.15. FIG. 2B illustrates an example of a one-frame synthesized imageobtained by modifying and combining four camera images.

[0067]FIG. 2C is an example of an image synthesis parameter group, inwhich respective sets of coordinates have one-to-one correspondence withthe coordinates of pixels in the synthesized image. Each set ofcoordinates includes parameters representing the camera number withwhich a camera image is taken and a two-dimensional array indicating thecoordinates of a pixel in the camera image used to generate the pixel inthe synthesized image corresponding to the set of coordinates. In theexample shown in FIG. 2C, information of “camera No. 1, coordinates (xi,yi)” is stored in the coordinates (xo, yo) in the image synthesisparameter group. This indicates that the pixel at the coordinates (xo,yo) in the synthesized image is generated by the pixel at thecoordinates (xi, yi) in the camera image from the camera 1. In this way,it is possible to describe the correspondence between the pixels in aplurality of camera images and those in a synthesized image by use of animage synthesis parameter group constructed as shown in FIG. 2C.

[0068] In this embodiment, it is assumed that the parameter storagesection 109 stores a plurality of image synthesis parameter groupshaving different spatial or temporal resolutions for at least either thecamera images or the synthesized image.

[0069] As such a plurality of image synthesis parameter groups havingdifferent spatial or temporal resolutions, assume specifically that theparameter storage section 109 stores a frame-base image synthesisparameter group and a field-base image synthesis parameter group havingthe same composition of the synthesized image. In the frame-base imagesynthesis parameter group, a one-frame camera image corresponds to aone-frame synthesized image. In the field-base image synthesis parametergroup, the first field of a camera image corresponds to one field of asynthesized image, while the second field of the camera imagecorresponds to the other field of the synthesized image. Thefield-to-field correspondence between the camera image and thesynthesized image can be easily described by imposing a restriction thatthe part of a camera image having an even y coordinate value correspondsto the part of a synthesized image having an even y coordinate value andthat the part of the camera image having an odd y coordinate valuecorresponds to the part of the synthesized image having an odd ycoordinate value.

[0070]FIG. 3 shows an example of the operation of the parameterselection section 108. Referring to FIG. 3, the parameter selectionsection 108 selects the image synthesis parameter group according to thevehicle speed detected by the vehicle motion detection section 107 basedon predetermined criteria. Specifically, the parameter selection section108 selects the frame-base image synthesis parameter group when themotion of the vehicle is relatively slow, for example, when the vehiclespeed is less than 10 km/h, and selects the field-base image synthesisparameter group when the motion of the vehicle is relatively fast, forexample, when the vehicle speed is 10 km/h or more. The image synthesissection 104 reads pixel data of camera images from the frame memories103 according to the camera numbers and the coordinate values in theimage synthesis parameter group selected by the parameter selectionsection 108, to generate a synthesized image. The generated synthesizedimage is converted to analog signals by the D/A converter 105 anddisplayed on the display 106.

[0071] By the operation as described above, images are combined everyfield when the vehicle speed is high and the motion in the camera imageis large. This enables generation of a natural synthesized image freefrom comb-shaped displacement. On the contrary, images are composedevery frame when the vehicle speed is low. This enables generation of aclear synthesized image with high resolution. That is, by changing theimage synthesis method according to the motion of the vehicle, it ispossible to reduce the unnaturalness of the synthesized image that isotherwise observed when the vehicle is moving, and also prevent thereduction in the resolution of the synthesized image that is otherwiseobserved when the vehicle is substantially standing still.

[0072] In this embodiment, the vehicle motion detection section 107detects the motion of the vehicle, represented by the rotational speedof the wheels, the driving direction, and the like, from signals sentfrom sensors placed on an axle, the shift lever, and the like.Alternatively, the vehicle motion may be detected from camera images.

[0073] In the case of detecting the vehicle motion from camera images,images from one of the cameras 101 are sequentially read from the framememory 103, to obtain an image composed of a plurality of temporallycontinuous frames or fields and calculate a motion vector in the image.Based on the calculated motion vector and camera parameters such as thepre-measured position, direction, and focal distance of the camera 101,the direction and speed of the motion of the road surface in the imageare determined and output as the vehicle speed.

[0074] By providing the vehicle motion detection section 107 thatdetects the vehicle motion from camera images as described above, it isno more required to supply inputs other than the images, such as thesignals from sensors placed on an axle and the shift lever. Therefore, asimple system construction can be realized.

[0075] The cameras 101 may have a function of switching a capturepattern according to an input switch signal. In such a case, theparameter selection section 108 may be constructed to switch the capturepattern of the cameras 101, in addition to selecting the image synthesisparameter group as described above, according to the vehicle motiondetected by the vehicle motion detection section 107.

[0076]FIG. 4 is a view showing an example of the operation of theparameter selection section 108 in the case described above. In thisexample, it is assumed that the cameras 101 permit switching betweeninterlaced imaging (one field every {fraction (1/60)} second) andnon-interlaced imaging (one frame every {fraction (1/30)} second) andswitching of the shutter speed (exposure time: {fraction (1/30)} second,{fraction (1/60)} second, and {fraction (1/120)} second).

[0077] As shown in FIG. 4, the parameter selection section 108 selectsthe image synthesis parameter group according to the vehicle speeddetected by the vehicle motion detection section 107 based onpredetermined criteria, and also switches the capture pattern of eachcamera 101. Specifically, the parameter selection section 108 selectsthe frame-base image synthesis parameter group when the vehicle speed isless than 10 km/h, or the field-base image synthesis parameter groupwhen it is 10 km/h or more. In addition, the parameter selection section108 switches the capture pattern of each camera 101 so thatnon-interlaced imaging and {fraction (1/30)}-second exposure time areselected when the vehicle speed is less than 10 km/h, interlaced imagingand {fraction (1/60)}-second exposure time are selected when the vehiclespeed is between 10 km/h or more and less than 30 km/h, and interlacedimaging and {fraction (1/120)}-second exposure time are selected whenthe vehicle speed is 30 km/h or more.

[0078] By the operation described above, when the vehicle speed is lessthan 10 km/h, images are taken by non-interlaced imaging and combinedper frame. This increases the resolution of the synthesized image. Whenthe vehicle speed is 10 km/h or more, images are taken by interlacedimaging and combined per field. This decreases the resolution, butnatural motion is obtained. In addition, by switching the exposure timewith the vehicle speed, a natural synthesized image with reduction inblurring due to the motion is obtained. In other words, a synthesizedimage with optimum quality determined by the motion of the vehicle canbe generated by the combination of the selection of the image synthesisparameter group and the switching of the capture pattern.

[0079] The construction shown in FIG. 1 includes the vehicle statusdetection section 200. The outputs of the vehicle status detectionsection 200 can be used in the selection of the image synthesisparameter group and the switching of the capture pattern, to furtherimprove the quality of the synthesized image. The vehicle statusdetection section 200, which is constructed of an optical sensor placedon the vehicle and the like, includes a means 210 for detecting thebrightness in the surroundings of the vehicle, a means 220 for detectingthe switching operation by the driver, and a means 230 for detecting anobstacle in the surroundings of the vehicle.

[0080]FIG. 5 is a view showing an example of the operation of theparameter selection section 108 using the output of the brightnessdetection means 210. In general, the exposure time required to takeone-frame image by interlaced imaging is half that required bynon-interlaced imaging. In the case of a moving image, a longer exposuretime causes blurring but provides a bright image. On the contrary, ashorter exposure time reduces blurring but darkens the image. Based onthis fact, the parameter selection section 108 performs the selection ofthe image synthesis parameter group and the switching of the capturepattern of each camera 101 in consideration of the brightness in thesurroundings of the vehicle detected by the brightness detection means210, in addition to the vehicle speed detected by the vehicle detectionsection 107. This enables generation of a synthesized image with optimumquality determined by the brightness in the surroundings of the vehicleand the vehicle speed.

[0081]FIG. 6 is a view showing an example of the operation of theparameter selection section 108 using the outputs of the switchdetection means 220 and the obstacle detection means 230. Assume thatthe parameter storage section 109 stores frame-base and field-base imagesynthesis parameter groups for two types of compositions shown in FIGS.7A and 7B, and that the driver is allowed to select one of threeoptions, “composition A”, “composition B”, and “automatic”, by switchoperation. The switch detection means 220 detects the switch operationby the driver. The obstacle detection means 230 measures the distancefrom an obstacle using an ultrasonic obstacle sensor placed on thevehicle and outputs the results.

[0082] The parameter selection section 108 selects a suitable imagesynthesis parameter group according to the detected status of thevehicle, that is, the selection by the driver and the distance from anobstacle, and the vehicle speed detected by the vehicle motion detectionsection 107.

[0083] Composition A of FIG. 7A includes a wider range of thesurroundings of the vehicle, while composition B of FIG. 7B shows anarrower range of the surroundings of the vehicle in an enlarged view.Therefore, supposing the vehicle speed is the same, greater comb-shapeddisplacement is generated in composition B than in composition A. Inview of this fact, the vehicle speed with which the image synthesisparameter group is switched from the frame base to the field base may bechanged according to the motion and status of the vehicle, as shown inFIG. 6. This enables generation of a synthesized image with furtheroptimum quality.

[0084] The vehicle status detection section 200 may also detect theoperations of the shift lever, the winker, and the like by the driver.

[0085] In this embodiment, the field-base and frame-base image synthesisparameter groups were used as an example. It is also possible to useother types of image synthesis parameter groups that are different inspatial or temporal resolution relation from each other. For example,when a one-frame camera image is constructed of a plurality of imagestaken for different durations, it is possible to use a plurality ofimage synthesis parameter groups having different spatial or temporalresolution combinations, to provide substantially the same effects asthose described above.

[0086] It should be noted that the criteria for selection of a parametergroup shown in FIGS. 3 to 6 and 10 are mere examples and that aparameter group may be selected according to the motion and status ofthe vehicle based on criteria other than the exemplified ones.

[0087] In this embodiment, field-base and frame-base image synthesisparameter groups were prepared. Alternatively, only a frame-base imagesynthesis group may be prepared to realize the operation describedabove. That is, a field-base image synthesis parameter group can beprepared by rounding the y coordinate value yi in a camera image to anodd or even number according to the field to which the y coordinatevalue yo in the frame-base image synthesis parameter group belongs.

[0088] Therefore, only a frame-base image synthesis parameter group maybe stored in the parameter storage section 109. When a frame-base imagesynthesis parameter group is required, the parameter selection section108 reads the frame-base image synthesis parameter group as it is. Whena field-base image synthesis parameter group is required, the parameterselection section 108 may convert the frame-base image synthesisparameter group stored in the parameter storage section 109 to afield-base image synthesis parameter group in the manner describedabove.

EMBODIMENT 2

[0089]FIG. 8 illustrates a construction of a vehicle surroundingsmonitoring system of EMBODIMENT 2 of the present invention. Thisconstruction is different from the construction shown in FIG. 1 in thata filtering section 310 is provided between the A/D converters 102 andthe frame memories 103 to perform filtering of camera images. Aparameter storage section 330 stores a plurality of sets of an imagesynthesis parameter group representing the correspondence between cameraimages and a synthesized image and a filter parameter groupcorresponding to the image synthesis parameter group. A parameterselection section 320 selects a set of the image synthesis parametergroup and the filter parameter group from the parameter storage section330 according to the vehicle motion detected by the vehicle motiondetection section 107 and the vehicle status detected by the vehiclestatus detection section 200. The filtering section 310 performsfrequency-band limitation filtering for respective camera imagesaccording to the filter parameter group in the set selected by theparameter selection section 320.

[0090]FIG. 9 shows an example of the filter parameter group, whichspecifies the cut-off frequency for each camera. The filtering section310 performs low-pass filtering with the cut-off frequency specified inthe filter parameter group for digitized image data output from the A/Dconverters 102.

[0091] In the illustrated example, the cut-off frequency is provided fora pixel clock fs, and “½ fs” is specified when the input image data isto be output as it is and “¼ fs” is specified when the input image datais to be filtered with a cut-off frequency of ¼ fs. The cut-offfrequency is calculated in advance based on a sampling theorem accordingto the contraction rate of the portion of the camera image that is mostcontracted during image synthesis according to the image synthesisparameter group. For example, when the contraction rate of the portionof a camera image from the camera 1 that is most contracted during imagesynthesis is ½ in terms of the length, the cut-off frequency for thecamera 1 is set at ¼ fs.

[0092] The contraction rate of a camera image can be determined usingthe coordinate values for the camera image in an image synthesisparameter group. For example, assume that the parameter element atcoordinates (x0, y0) in an image synthesis parameter group includes“camera No. 1, coordinates (x1, y1)”, and the parameter element atcoordinates (x0+1, y0) in the image synthesis parameter group includes“camera No. 1, coordinates (x2, y2)”. In this case, when the distancebetween the coordinates (x1, y1) and the coordinates (x2, y2) is D, theportions at and around the coordinates (x1, y1) and the coordinates (x2,y2) in the camera image from the camera 1 have been contracted to 1/D inthe synthesized image. In this way, it is possible to obtain thecontraction rates at and around certain pixels in a camera image. Theminimum of the thus-obtained contraction rates is the contraction rateof the most contracted portion of the camera image.

[0093] The parameter selection section 320 selects the image synthesisparameter group and the filter parameter group according to the vehiclespeed detected by the vehicle motion detection section 107 and thevehicle status detected by the vehicle status detection section 200 asshown in FIG. 10, for example. The other operation is substantially thesame as that described in EMBODIMENT 1.

[0094] As described in relation to the problems to be solved, when animage is contracted in modification, aliasing distortion may begenerated in a synthesized image unless a high-range frequency componentis removed according to the contraction rate. In this embodiment, suchaliasing distortion can be eliminated by performing the low-passfiltering according to the filter parameter group. Moreover, in the caseof switching the composition of a synthesized image as shown in FIG. 7,the filter parameter group may be switched together with the switchingof the composition. This enables generation of a synthesized image withoptimum quality with reduced aliasing distortion determined by thestatus and speed of the vehicle.

[0095]FIGS. 11A and 11B are views for description of another example ofthe filter parameter group. The filter parameter group shown in FIG. 11Bincludes parameter elements having one-to-one correspondence with thecoordinates of pixels in a one-frame camera image. Each parameterelement specifies the cut-off frequency for filtering of each cameraimage. In other words, the filter parameter group includes filteringsetting data for respective pixel positions of each camera image.

[0096] The parameter selection section 320 selects the image synthesisparameter group and the filter parameter group according to the vehiclespeed detected by the vehicle detection section 107 and the vehiclestatus detected by the vehicle status detection section 200, and outputsthe selected results while performing sequential scanning. The filteringsection 310 performs low-pass filtering for the digitized image dataoutput from the A/D converters 102 according to the cut-off frequencyinformation in the filter parameter group output from the parameterselection section 320.

[0097] In the filter parameter group shown in FIG. 9, only one cut-offfrequency is provided for one camera image. This is not necessarilysuitable when the contraction rate varies with the positions on a cameraimage in the image synthesis. On the contrary, in the filter parametergroup shown in FIG. 11B, the cut-off frequency can be changed with thepositions on a camera image, and thus the non-contracted portion of theinput image is prevented from blurring.

[0098] The filter parameter group shown in FIG. 11B includes filteringsetting data having one-to-one correspondence with the coordinates ofpixels in a one-frame camera image. In general, filter parameters ofadjacent pixels tend to have similar values. Therefore, common settingdata may be held for a plurality of adjacent pixels. For example, commonsetting data may be held every rectangular area of 8×8 pixels. By thissetting, in addition to preventing the non-contracted portion of theimage from blurring, the data amount of the filter parameter group canbe reduced compared with the case of holding filtering setting dataevery pixel.

[0099] A more significant effect can be provided by combining theconstructions of EMBODIMENT 1 and EMBODIMENT 2. That is, the interlacenoise can be eliminated by the technical feature of EMBODIMENT 1 and thealiasing distortion can be eliminated by the technical feature ofEMBODIMENT 1. This enables generation of a synthesized image with goodquality even when the vehicle is moving at high speed.

EMBODIMENT 3

[0100]FIG. 12 illustrates a construction of a vehicle surroundingsmonitoring system of EMBODIMENT 3 of the present invention. Thisconstruction is different from the construction shown in FIG. 1 in thata brightness correction section 410 is provided between the framememories 103 and the image synthesis section 104 to correct thebrightness and tint of camera images. A brightness correction parametercalculation section 420 is also provided to calculate brightnesscorrection parameters used by the brightness correction section 410.

[0101]FIGS. 13A to 13D illustrate an example of an image synthesisparameter group for generating a synthesized image from a plurality ofcamera images. FIGS. 13A and 13B are examples of images taken with twocameras placed on the rear of the vehicle. In the illustrated example,an image of the area-just behind the vehicle is taken doubly by cameras1 and 2. FIG. 13C shows an example of a synthesized image, and FIG. 13Dshows an example of an image synthesis parameter group.

[0102] In the example shown in FIG. 13D, the parameter element at thecoordinates (xo, yo) in the image synthesis parameter group indicatesthat a synthesized image should be generated by combining the pixel atthe coordinates (Xi₁, Yi₁) in the image from the camera 1 and the pixelat the coordinates (Xi₂, Yi₂) in the image from the camera 2 at aweighting ratio of 0.7:0.3. The image synthesis section 104 assignsweights to the respective camera images and then adds, to sequentiallyoutput the results as a synthesized image. This enables generation of asmooth synthesized image free from formation of a conspicuous juncturedue to difference in brightness and tint between the camera images,compared with the case of generating each pixel of a synthesized imagefrom a single pixel of one camera image.

[0103] Moreover, in this embodiment, brightness correction parametersare calculated using information on an overlap area on a synthesizedimage.

[0104] Specifically, the image synthesis parameter group stored in theparameter storage section 109 includes overlap area data indicating thecoordinates of a pixel in a camera image that corresponds to an overlaparea on a synthesized image. The brightness correction parametercalculation section 420 receives the overlap area data included in theimage synthesis parameter group and calculates a brightness correctionparameter using brightness and tint data for the pixel in the cameraimage corresponding to the overlap area indicated by the overlap areadata.

[0105]FIGS. 14A and 14B illustrate an example of a synthesized imageincluding an overlap area and an example of an image synthesis parametergroup for this synthesized image. In the synthesized image of FIG. 14A,the hatched portions represent the coverages of cameras 1 and 2, whichoverlap at the rear of the vehicle forming an overlap area. The imagesynthesis parameter group of FIG. 14B includes overlap area dataindicating the coordinates of pixels in the camera images correspondingto the overlap area, as partly shown in FIG. 14B. That is, as theoverlap area data it is shown that the pixel at the coordinates (Xi1,Yi1) in the image from camera 1 and the pixel at the coordinates (Yi2,Yi2) in the image from camera 2 correspond to the coordinates (xo, yo)in the overlap area.

[0106] It is possible to determine whether a given position is inside anon-overlap area or an overlap area depending on whether the weightingratio is “1:0” or otherwise. Alternatively, a given position may bedetermined to be inside an overlap area if two or more effective cameranumbers are specified, and inside a non-overlap area if only oneeffective camera is specified.

[0107] The parameter selection section 108 selects the image synthesisparameter group according to the vehicle speed detected by the vehiclemotion detection section 107 and the vehicle status detected by thevehicle status detection section 200 based on predetermined criteria,sequentially reads the selected parameters by interlaced scanning, andoutputs the results to the image synthesis section 104. The overlap areadata in the image synthesis parameter group output from the parameterselection section 108 is also output to the brightness correctionparameter calculation section 420.

[0108] The brightness correction parameter calculation section 420determines the distributions of brightness and tint of the respectivecamera images in the overlap area, from the overlap area data outputfrom the parameter selection section 108 and the pixel values read fromthe frame memories 103 by the image synthesis section 104, to calculatebrightness correction parameters with which the distributions ofbrightness and tint of the respective camera images can substantiallymatch with each other in the overlap area.

[0109] The brightness correction section 410 corrects the brightness forthe pixel values read from the frame memories 103 by the image synthesissection 104 according to the brightness correction parameters calculatedby the brightness correction parameter calculation section 420, andoutput the results to the image synthesis section 104. The imagesynthesis section 104 generates a synthesized image from the cameraimages of which brightness has been corrected by the brightnesscorrection section 410, according to the image synthesis parameter groupoutput from the parameter selection section 108.

[0110] A procedure of calculating the brightness correction parameterswill be described in a specific example.

[0111] Assume that the brightness correction parameter calculationsection 420 outputs as the brightness correction parameters a gaincoefficient and an offset coefficient with which the averages anddistributions of the brightness of respective camera images can matchwith each other in the overlap area. Also assume that the brightnesscorrection section 410 corrects the pixel values of the respectivecamera images with a linear equation using the gain coefficient and theoffset coefficient as the brightness correction parameters.

[0112] Assume that a plurality of overlap areas exist. For example,assume that there exist four overlap areas, a first overlap area formedby cameras 1 and 2, a second overlap area formed by cameras 1 and 3, athird overlap area formed by cameras 2 and 4, and a fourth overlap areaformed by cameras 3 and 4.

[0113] First focusing on the first overlap area, the brightnesscorrection parameter calculation section 420 accumulates images takenwith the cameras 1 and 2 read from the frame memories 103 by the imagesynthesis section 104 by one frame each, and obtains the averages anddistributions of the brightness of the images taken with the cameras 1and 2 in the first overlap area. Likewise, for the second to fourthoverlap areas, the averages and distributions of the brightness ofcamera images in each of the overlap areas are obtained.

[0114] Next, as brightness correction parameters, a gain and an offsetare determined so that the averages and distributions of the brightnessof the images from the cameras 1 and 2 in the first overlap area matchwith each other. For example, consider the case that the average anddistribution of the brightness of the image from the camera 1 are 100and 10, respectively, and the average and distribution of the brightnessof the image from the camera 2 are 121 and 11, respectively. Then, theaverages and distributions of the images from the cameras 1 and 2 willmatch with each other when the gain and the offset are set to 1 and 0respectively for the image from the camera 1, and 10/11 and −10respectively for the image from the camera 2.

[0115] Using the brightness correction parameters determined for thefirst overlap area, the average and distribution of the brightness ofthe image from the camera 1 in the second overlap area are corrected.Note that in this example where the gain is 1 and the offset is 0 forthe image from the camera 1, no correction is required for the averageand distribution of the brightness of the image from the camera 1.Brightness correction parameters for an image from the camera 3 are thendetermined so that the averages and distributions of the brightness ofthe images from the cameras 1 and 3 match with each other. Likewise, theaverage and distribution of the brightness of the image from the camera2 in the third overlap area are corrected using the brightnesscorrection parameters determined in the first overlap area. Brightnesscorrection parameters for an image from the camera 4 are then determinedso that the averages and distributions of the brightness of the imagesfrom the cameras 2 and 4 match with each other.

[0116] In the calculation procedure described above, proper brightnesscorrection may not be obtained in the fourth overlap area. In view ofthis, as another calculation procedure, brightness correction parametersmay be calculated temporarily in the respective overlap areas. For acamera image existing in a plurality of overlap areas, the average ofthe temporarily calculated brightness correction parameters may bedetermined as the final brightness correction parameters for the cameraimage. This procedure fails to completely match the brightness and thetint between camera images in each overlap area. However, as the entiresynthesized image, it is expected that all of the overlap areas have nogreat displacement in brightness and tint.

[0117] When a plurality of overlap areas exist, the priorities given tothe overlap areas to be followed during the calculation of brightnesscorrection parameters may be determined according to the motion of thevehicle detected by the vehicle motion detection section 107.

[0118] For example, when the vehicle is moving backward, a high prioritymay be given to an overlap area located at the rear of the vehicle onthe synthesized image. When the vehicle is moving forward, a highpriority may be given to an overlap area located at the front of thevehicle on the synthesized image. In general, the driver tends to payattention to the rear of the vehicle when driving backward and to thefront of the vehicle when driving forward. Therefore, by givingpriorities, the brightness and the tint can be made uniform in theoverlap area at the position to which the driver tends to pay attention,although they may not be uniform in other overlap areas. The resultantsynthesized image is therefore more natural for the driver.

[0119] As described above, in this embodiment, brightness correctionparameters are calculated using data on the brightness and tint ofpixels in camera images corresponding to an overlap area, and thebrightness is corrected using the parameters. This enables generation ofa smooth synthesized image free from formation of a conspicuous juncturedue to difference in brightness and tint between the camera images.

[0120] The construction of FIG. 12 includes the brightness correctionsection 410. However, if the cameras 101 incorporate therein a functioncorresponding to that of the brightness correction section 410, thebrightness correction section 410 may be omitted and thus the systemconstruction can be simplified.

[0121] The technique for brightness correction is not limited to thatdescribed in this embodiment, but other techniques may also be used aslong as the brightness and tint of camera images in an overlap area canbe made close to each other, and substantially the same effect as thatdescribed in this embodiment can be attained.

[0122] In this embodiment, weighting information in the image synthesisparameter group was used to identify an overlap area. Alternatively, inplace of the weighting information, some exclusive informationindicating whether or not a given position is inside an overlap area maybe included in the image synthesis parameter group.

EMBODIMENT 4

[0123] In EMBODIMENT 4 of the present invention, in an overlap area ofthe coverages of a plurality of cameras, the selection of the cameraimages used for image synthesis and the setting of weights for therespective camera images are appropriately changed.

[0124] The construction of the vehicle surroundings monitoring system ofthis embodiment is substantially the same as that shown in FIG. 1,except that the parameter storage section 109, which is the same inconstruction, includes a plurality of image synthesis parameter groupsdifferent in the selection of camera images used for image synthesis inan overlap area.

[0125]FIG. 15 is a view showing the positions of cameras and obstaclesensors as the obstacle detection means 230 on a vehicle in thisembodiment. Referring to FIG. 15, cameras 1 and 2 are placed on pillarsat the rear of the vehicle, where the camera 1 takes images on the leftside of the rear area of the vehicle, while the camera 2 takes images onthe right side thereof. The coverages of the cameras 1 and 2 overlap inthe rear of the vehicle. Two obstacle sensors are placed on the rear ofthe vehicle to enable detection of an obstacle in the rear area of thevehicle.

[0126]FIGS. 16A to 16C are views illustrating examples of a synthesizedimage and image synthesis parameter groups for generation of thesynthesized image. FIG. 16A illustrates a synthesized image, and FIGS.16B and 16C illustrate two image synthesis parameter groups A and B forgeneration of the synthesized image of FIG. 16A. The image synthesisparameter group A (FIG. 16B) and the image synthesis parameter group B(FIG. 16C) are used to generate the same synthesized image incomposition (FIG. 16A), but use a different camera image for imagesynthesis in the overlap area in the rear of the vehicle. The imagesynthesis parameter group A uses an image from the camera 1 while theimage synthesis parameter group B uses an image from the camera 2.

[0127] The operation of the vehicle surroundings monitoring system ofthis embodiment will be described.

[0128] When the obstacle detection means 230 detects an obstacle on theright or left side of the rear area of the vehicle, it outputs theposition of the detected obstacle to the parameter selection section108. The parameter selection section 108 selects an image synthesisparameter group according to the position of the detected obstacle andthe switch operation by the user.

[0129] For example, the image synthesis parameter group A is selectedwhen the obstacle exists on the left side of the rear area, and theimage synthesis parameter group B is selected when the obstacle existson the right side of the rear area. The image synthesis section 104reads camera images from the frame memories 103 according to theselected image synthesis parameter group to generate a synthesizedimage.

[0130] By the operation described above, in the case shown in FIG. 17A,for example, where an obstacle such as another vehicle exists on theleft side of the rear area, the image synthesis parameter group A isselected. This makes the juncture formed between the camera images lessconspicuous on the resultant synthesized image as shown in FIG. 17B. Ifthe image synthesis parameter group B is selected, the juncture betweenthe camera images will run across the obstacle vehicle on thesynthesized image, making the vehicle unnatural with the juncture asshown in FIG. 17C. In other words, by selecting the image synthesisparameter group according to the position of an obstacle, the existenceof the juncture can be made less conspicuous on the synthesized image,and thus a synthesized image with natural appearance can be generated.

[0131] The parameter selection section 108 may otherwise switch theimage synthesis parameter group according to the motion of the vehicledetected by the vehicle motion detection section 107, in place of theoutput from the vehicle status detection section 200. By switching inthis manner, the camera image used for image synthesis in the overlaparea can be switched according to the direction and speed of the drivingof the vehicle. Therefore, by selecting a camera image from a camera ofwhich coverage is closer to the forward area of the vehicle, forexample, it is possible to reduce formation of a juncture between cameraimages in and around an area mostly viewed by the driver, and thusdisplay a synthesized image with which the driver can feel easy indriving. Naturally, it is also possible to use both the outputs from thevehicle status detection section 200 and the vehicle motion detectionsection 107.

[0132]FIG. 18 illustrates another example of the image synthesisparameter group in this embodiment. The image synthesis parameter groupof FIG. 18 includes a plurality of sets of weighting information forcamera images in an overlap area on a synthesized image. For example,the position at the coordinates (x0, y0) in the image synthesisparameter group is a position at which a pixel at the coordinates (Xi1,Yi1) in a camera image from the camera 1 and a pixel at the coordinates(Xi2, Yi2) in a camera image from the camera 2 overlap with each other.This position includes two sets of weighting information, A (0.7:0.3)and B (0.2:0.8) as the weighting ratio used during synthesis.

[0133] The parameter selection section 108 selects one of a plurality ofsets of weighting information included in the image synthesis parametergroup according to the output of the vehicle status detection section200. For example, the weighting information A is selected when anobstacle exists on the left side of the rear area of the vehicle, andthe weighting information B is selected when an obstacle exists on theright side.

[0134] Thus, by selecting weighting information for image synthesisaccording to the position of an obstacle, it is possible to reduce theunnaturalness of the juncture formed between the camera images on asynthesized image and thus generate a synthesized image with a smootherjuncture.

[0135]FIG. 19 illustrates yet another example of the image synthesisparameter group in this embodiment. The image synthesis parameter groupof FIG. 19 is equivalent to that of FIG. 18 except that one weightreference number indicating a combination of sets of weightinginformation is given, in place of sets of weighting information, foreach pixel.

[0136] In the example shown in FIG. 19, the weight reference number forthe coordinates (x0, y0) in the image synthesis parameter group is “3”.By referring to another table, it is found that the weight referencenumber “3” indicates that the weighting ratio of the pixels to becombined is 0.7:0.3 (weighting information A) or 0.2:0.8 (weightinginformation B).

[0137] While the image synthesis parameter group of FIG. 18 needs tohold four values as weighting information for each coordinate position,the image synthesis parameter group of FIG. 19 can hold only one weightreference information value. This indicates that the data amount of theimage synthesis parameter group of FIG. 19 is smaller as the number ofpixels in the image synthesis parameter group is greater or the numberof combinations of sets of weighting information is smaller.

[0138] Thus, the data amount of the image synthesis parameter group canbe reduced by holding the weight reference number indicating acombination of a plurality of sets of weighting information, in place ofholding a plurality of sets of weighting information, for eachcoordinate position in the image synthesis parameter group.

[0139]FIGS. 20A and 20B illustrate other examples of the image synthesisparameter groups in this embodiment, which are generated using adithering method based on the image synthesis parameter group of FIG.18.

[0140] The image synthesis parameter group A of FIG. 20A is generated bya so-called dithering method, where, based on the camera numbers and theweighting information A for each pixel in the image synthesis parametergroup of FIG. 18, one of the two camera images is selected for eachpixel in the vicinity of a given pixel so that the ratio of the numberof pixels using one camera image to that using the other camera image isclose to the specified weighting ratio. Likewise, the image synthesisparameter group B of FIG. 20B is generated based on the camera numbersand the weighting information B for each pixel in the image synthesisparameter group of FIG. 18.

[0141] In the weighting information A in the image synthesis parametergroup of FIG. 18, at the coordinates (x0, y0), the weights are 0.7 forthe camera 1 and 0.3 for the camera 2. In the image synthesis parametergroup A of FIG. 20A, the camera number is selected so that the ratio ofthe number of pixels using the camera 1 to that using the camera 2 isclose to 0.7:0.3 in the vicinity of the pixel at the coordinates (x0,y0). Thus, in the example shown in FIG. 20A, although the camera 1 isselected for the coordinates (x0, y0), the ratio of the number of pixelsusing the camera 1 to that using the camera 2 is about 0.7:0.3 at andaround (x0, y0).

[0142] The dithering method described above is widely known anddisclosed in “Television image information engineering hand-book”, ed.by The Institute of Image Information and Television Engineers, Ohmsha,Ltd., and the like. Detailed description on the method for selectingcamera images is therefore omitted here.

[0143] The parameter selection section 108 selects the image synthesisparameter group A or B according to the output of the vehicle statusdetection section 200.

[0144] Thus, by selecting one from a plurality of dithered imagesynthesis parameter groups according to the position of an obstacle forimage synthesis, it is possible to reduce the unnaturalness of thejuncture formed between the camera images on a synthesized image andthus generate a synthesized image with a smoother juncture.

[0145] The total data amount of the image synthesis parameter group A ofFIG. 20A and the image synthesis parameter group B of FIG. 20B issmaller than the data amount of the image synthesis parameter group ofFIG. 18 by the amount of weighting information.

[0146] Moreover, when image synthesis is performed using the imagesynthesis parameter group A of FIG. 20A or the image synthesis parametergroup B of FIG. 20B, the number of pixels read from the frame memories103 by the image synthesis section 104 is smaller, and the amount ofcalculation by the image synthesis section 104 is smaller by theaddition of the weighting, compared with the image synthesis using theweight-added image synthesis parameter group of FIG. 18.

[0147] Thus, the use of the image synthesis parameter group generated bythe dithering method provides the effect of reducing the processingamount required for image synthesis, in addition to the effect ofreducing the data amount of the image synthesis parameter group,compared with the use of the weighting information-added image synthesisparameter.

[0148] In the embodiments described above, the display 106 displaysinterlaced scanning images, and the parameter selection section 108, 320reads an image synthesis parameter group by interlaced scanning. Thesame effects as those described in the respective examples can also beobtained by adopting non-interlaced scanning in both operations.

[0149] In the embodiments described above, each image synthesisparameter group is data in the form of a table indicating thecorrespondence between the coordinates in camera images and thecoordinates in a synthesized image. Alternatively, it may be describedin other forms such as a mathematical expression including variables anda program as long as the correspondence between camera images and asynthesized image is clarified. When such forms are adopted, the imagesynthesis section may be constructed to obtain corresponding coordinatesusing an image synthesis parameter group described in the form of amathematical expression including variables or a program. JapanesePatent Application No. 10-217261, for example, discloses a method forgenerating image synthesis parameter groups described in the forms of amathematical expression including variables and a program.

[0150] In the embodiments described above, it was assumed that the imagesynthesis parameter group included elements having one-to-onecorrespondence with the coordinates of pixels in a one-frame synthesizedimage and had the same size as that of the synthesized image. The sizeof the image synthesis parameter group is not necessarily the same asthat of the synthesized image. For example, an image synthesis parametergroup having a size larger than the synthesized image may be prepared,and a portion having the same size as the synthesized image may be cutout from this image synthesis parameter group to be used for imagesynthesis.

[0151] In the above embodiments, the monitoring system and the imageprocessor of the present invention were described as being applied tocars. Alternatively, they may be applied to other types of vehicles suchas shipping and airplanes. Otherwise, cameras may be placed on anon-mobile object to be monitored such as a store, a house, a showroom,and the like.

[0152] The positions of a plurality of cameras and the number of camerasare not limited to those described above.

[0153] The function of the image processor of the present invention maybe implemented by hardware entirely or partially using exclusiveequipment, or may be implemented by software. It is also possible to usea recording medium or a transmission medium storing a program programmedto enable a computer to execute the entire or partial function of theimage processor of the present invention. For example, a computer asshown in FIG. 21 may be used, where the processing means such as theimage synthesis section and the parameter selection section may beimplemented by a software program executed by a CPU 510 and the resultsmay be stored in a ROM 520 or a RAM 530.

[0154] Thus, according to the present invention, it is possible toreduce the unnaturalness of a synthesized image observed when thevehicle is moving and also prevent the reduction in resolution observedwhen the vehicle is standing still. In addition, aliasing distortion ona synthesized image can be effectively suppressed. Moreover, it ispossible to reduce the unnaturalness of the juncture formed betweencamera images on a synthesized image.

[0155] While the present invention has been described in a preferredembodiment, it will be apparent to those skilled in the art that thedisclosed invention may be modified in numerous ways and may assume manyembodiments other than that specifically set out and described above.Accordingly, it is intended by the appended claims to cover allmodifications of the invention that fall within the true spirit andscope of the invention.

What is claimed is:
 1. An image processor for receiving camera imagestaken with a plurality of cameras capturing the surroundings of avehicle and generating a synthesized image from the camera images, theimage processor comprising: a parameter generation section constructedto be able to generate a plurality of image synthesis parameter groupseach representing the correspondence between the camera images and thesynthesized image and having different spatial or temporal resolutionrelations, wherein the synthesized image is generated from the cameraimages according to the image synthesis parameter group generated by theparameter generation section, and the parameter generation sectionswitches the image synthesis parameter group to be generated accordingto an output of a vehicle motion detection section for detecting themotion of the vehicle.
 2. The image processor of claim 1, wherein theparameter generation section comprises: a parameter storage section forstoring the plurality of image synthesis parameter groups, and aparameter selection section for selecting at least one among theplurality of image synthesis parameter groups stored in the parameterstorage section according to the output of the vehicle motion detectionsection.
 3. The image processor of claim 1, wherein the camera imagesare interlaced images, and the plurality of image synthesis parametergroups include at least a frame-base image synthesis parameter group anda field-base image synthesis parameter group.
 4. The image processor ofclaim 3, wherein the parameter generation section generates thefield-base image synthesis parameter group when the motion of thevehicle detected by the vehicle motion detection section is relativelyfast, and generates the frame-base image synthesis parameter group whenthe motion of the vehicle is relatively slow.
 5. The image processor ofclaim 1, wherein the vehicle motion detection section detects the motionof the vehicle from the camera images.
 6. The image processor of claim1, wherein the plurality of cameras are constructed to be able to switcha capture pattern according to an input switch signal, and the parametergeneration section sends the switch signal to the cameras, together withgenerating the image synthesis parameter group, according to the outputof the vehicle motion detection section, to switch the capture patternsof the cameras.
 7. The image processor of claim 6, wherein the parametergeneration section selects the image synthesis parameter group andswitches the capture patterns of the cameras according to an output of avehicle status detection section for detecting brightness of thesurroundings of the vehicle, in addition to the output of the vehiclemotion detection section.
 8. The image processor of claim 1, wherein theparameter generation section generates the image synthesis parametergroup according to an output of a vehicle status detection section fordetecting the vehicle status such as an operation by a driver of thevehicle or whether or not an obstacle exists in the surroundings of thevehicle, in addition to the output of the vehicle motion detectionsection.
 9. A monitoring system comprising: a plurality of cameras forcapturing the surroundings of a vehicle; an image processing section forreceiving camera images from the plurality of cameras and generating asynthesized image from the camera images; and a display section fordisplaying the synthesized image generated by the image processingsection, wherein the image processing section comprises a parametergeneration section constructed to be able to generate a plurality ofimage synthesis parameter groups each representing the correspondencebetween the camera images and the synthesized image and having differentspatial or temporal resolution relations, the synthesized image isgenerated from the camera images according to the image synthesisparameter group generated by the parameter generation section, and theparameter generation section generates at least one of the plurality ofimage synthesis parameter groups according to an output of a vehiclemotion detection section for detecting the motion of the vehicle.
 10. Animage processor for receiving camera images taken with a plurality ofcameras capturing the surroundings of a vehicle and generating asynthesized image from the camera images, the image processorcomprising: a parameter storage section for storing a plurality of setsof an image synthesis parameter group representing the correspondencebetween the camera images and the synthesized image and a filterparameter group corresponding to the image synthesis parameter group; aparameter selection section for selecting at least one among theplurality of sets of the image synthesis parameter group and the filterparameter group stored by the parameter storage section according to anoutput of a vehicle motion detection section for detecting the motion ofthe vehicle and an output of a vehicle status detection section fordetecting the status of the vehicle such as an operation by a driver ofthe vehicle or whether or not an obstacle exists in the surroundings ofthe vehicle; and a filtering section for performing frequency bandlimitation filtering for the camera images according to the filterparameter group of the set selected by the parameter selection section,wherein the synthesized image is generated from the camera imagesfiltered by the filtering section according to the image synthesisparameter group of the set selected by the parameter selection section.11. The image processor of claim 10, wherein the filter parameter groupincludes filtering setting data for each pixel position of the cameraimage.
 12. A monitoring system comprising: a plurality of cameras forcapturing the surroundings of a vehicle; an image processing section forreceiving camera images from the plurality of cameras and generating asynthesized image from the camera images; and a display section fordisplaying the synthesized image generated by the image processingsection, wherein the image processing section comprises: a parameterstorage section for storing a plurality of sets of an image synthesisparameter group representing the correspondence between the cameraimages and the synthesized image and a filter parameter groupcorresponding to the image synthesis parameter group; a parameterselection section for selecting at least one among the plurality of setsof the image synthesis parameter group and the filter parameter groupstored by the parameter storage section according to an output of avehicle motion detection section for detecting the motion of the vehicleand an output of a vehicle status detection section for detecting thestatus of the vehicle such as an operation by a driver of the vehicle orwhether or not an obstacle exists in the surroundings of the vehicle;and a filtering section for performing frequency band limitationfiltering for the camera images according to the filter parameter groupof the set selected by the parameter selection section, wherein thesynthesized image is generated from the camera images filtered by thefiltering section according to the image synthesis parameter group ofthe set selected by the parameter selection section.
 13. An imageprocessor for receiving camera images taken with a plurality of camerascapturing the surroundings of a vehicle and generating a synthesizedimage from the camera images, the image processor comprising: abrightness correction parameter calculation section for calculatingbrightness correction parameters for correcting the brightness and tintof the camera images; and a brightness correction section for correctingthe brightness and tint of the camera images using the brightnesscorrection parameters calculated by the brightness correction parametercalculation section, wherein the synthesized image is generated from theplurality of camera images subjected to brightness correction by thebrightness correction section according to an image synthesis parametergroup representing the correspondence between the camera images and thesynthesized image, the image synthesis parameter group includes overlaparea data for an overlap area on the synthesized image in whichcoverages of the plurality of cameras overlap, the overlap area dataindicating coordinates of pixels in camera images corresponding to theoverlap area, and the brightness correction parameter calculationsection receives the overlap area data and calculates the brightnesscorrection parameters using brightness and tint data for the pixels inthe camera images corresponding to the overlap area indicated by theoverlap area data.
 14. The image processor of claim 13, wherein thebrightness correction parameter calculation section performs statisticalprocessing on the brightness in the overlap area for the camera imagescorresponding to the overlap area, and calculates the brightnesscorrection parameters based on the processing results.
 15. The imageprocessor of claim 13, wherein, when a plurality of overlap areas exist,the brightness correction parameter calculation section sets prioritiesto the overlap areas to be considered during the calculation of thebrightness correction parameters according to an output of a vehiclemotion detection section for detecting the motion of the vehicle. 16.The image processor of claim 13, wherein the brightness correctionsection is incorporated in the cameras.
 17. A monitoring systemcomprising: a plurality of cameras for capturing the surroundings of avehicle; an image processing section for receiving camera images fromthe plurality of cameras and generating a synthesized image from thecamera images; and a display section for displaying the synthesizedimage generated by the image processing section, wherein the imageprocessing section comprises: a brightness correction parametercalculation section for calculating brightness correction parameters forcorrecting the brightness and tint of the camera images; and abrightness correction section for correcting the brightness and tint ofthe camera images using the brightness correction parameters calculatedby the brightness correction parameter calculation section, wherein thesynthesized image is generated from the plurality of camera imagessubjected to brightness correction by the brightness correction sectionaccording to an image synthesis parameter group representing thecorrespondence between the camera images and the synthesized image, theimage synthesis parameter group includes overlap area data for anoverlap area on the synthesized image in which coverages of theplurality of cameras overlap, the overlap area data indicatingcoordinates of pixels in camera images corresponding to the overlaparea, and the brightness correction parameter calculation sectionreceives the overlap area data and calculates the brightness correctionparameters using brightness and tint data for the pixels in the cameraimages corresponding to the overlap area indicated by the overlap areadata.
 18. An image processor for receiving camera images taken with aplurality of cameras capturing the surroundings of a vehicle andgenerating a synthesized image from the camera images, wherein, in anoverlap area in which coverages of a plurality of cameras overlap on thesynthesized image, a camera image used for generation of the synthesizedimage is selected among camera images from the plurality of camerasaccording to an output of a vehicle motion detection section fordetecting the motion of the vehicle or an output of a vehicle statusdetection section for detecting the status of the vehicle such as anoperation by a driver of the vehicle or whether or not an obstacleexists in the surroundings of the vehicle.
 19. An image processor forreceiving camera images taken with a plurality of cameras capturing thesurroundings of a vehicle and generating a synthesized image from thecamera images, wherein, in an overlap area in which coverages of aplurality of cameras overlap on the synthesized image, weights to cameraimages from the plurality of cameras are set according to an output of avehicle motion detection section for detecting the motion of the vehicleor an output of a vehicle status detection section for detecting thestatus of the vehicle such as an operation by a driver of the vehicle orwhether or not an obstacle exists in the surroundings of the vehicle.20. The image processor of claim 19, wherein the image processorincludes: an image synthesis parameter group associating pixels in thesynthesized image with pixels in the camera images; and a tablerepresenting the correspondence between a weight reference number and acombination of sets of weighting information, wherein a portion of theimage synthesis parameter group corresponding to the overlap area holdsany of the weight reference numbers shown in the table.
 21. A monitoringsystem comprising: a plurality of cameras for capturing the surroundingsof a vehicle; an image processing section for receiving camera imagesfrom the plurality of cameras and generating a synthesized image fromthe camera images; and a display section for displaying the synthesizedimage generated by the image processing section, wherein, in an overlaparea in which coverages of a plurality of cameras overlap on thesynthesized image, a camera image used for generation of the synthesizedimage is selected among camera images from the plurality of camerasaccording to an output of a vehicle motion detection section fordetecting the motion of the vehicle or an output of a vehicle statusdetection section for detecting the status of the vehicle such as anoperation by a driver of the vehicle or whether or not an obstacleexists in the surroundings of the vehicle.
 22. A monitoring systemcomprising: a plurality of cameras for capturing the surroundings of avehicle; an image processing section for receiving camera images fromthe plurality of cameras and generating a synthesized image from thecamera images; and a display section for displaying the synthesizedimage generated by the image processing section, wherein, in an overlaparea in which coverages of a plurality of cameras overlap on thesynthesized image, weights to camera images from the plurality ofcameras are set according to an output of a vehicle motion detectionsection for detecting the motion of the vehicle or an output of avehicle status detection section for detecting the status of the vehiclesuch as an operation by a driver of the vehicle or whether or not anobstacle exists in the surroundings of the vehicle.
 23. An imageprocessor for receiving camera images taken with a plurality of camerascapturing the surroundings of a vehicle and generating a synthesizedimage from the camera images, wherein the image processor includes animage synthesis parameter group associating pixels in the synthesizedimage with pixels in the camera images, and in an overlap area in whichcoverages of the plurality of cameras overlap on the synthesized image,the image synthesis parameter group includes weights set to cameraimages from the plurality of cameras represented by a dithering method.